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Abstract- Educational institutions are racing to move from conventional classrooms to online delivery of courses 
and learning materials. In many cases, the process of migration from the classroom to the web is simply taking the 
form format and content of the traditional teacher-centered model and delivering it electronically. In one university 
program however, the faculty and administrators made a conscious effort to take a broad, extensive approach to 
designing and implementing a fully online masters program. This approach was centered in a comprehensive needs 
assessment model and sought input from students, faculty, industry leaders, technical specialists, and other 
stakeholders. In addition to focus groups of students and industry professionals who might be expected to hire 
program graduates, .a Delphi process was developed and used with College faculty and administrators as a key 
element for the clarification of goals, expectations and objectives for the entire program, courses and student 
competencies. 



The Perceived Need 



The Conrad N Hilton College of Hotel and Restaurant Management is generally regarded as one of a handful of top 
programs in the world which prepares people to become managers in the hospitality field. The College was 
established in 1969 to meet the demand for professional training in the increasingly complex hospitality industry. I o 
satisfy this demand, the college seeks to prepare students for effective and profitable management roles in their 
choice of careers and to offer a cultural experience that will aid them in taking their places as productive members of 

society. 



Hotel and restaurant management requires a diversity of skills found in many disciplines including accounting 
computer science, economics, law, mathematics and psychology. For this reason, the college emphasizes broad 
information skills, flexibility in abstract problem solving and in-depth studies of specific skills. The curriculum is 
designed to prepare students to cope with changing business conditions and to present both theoretical and practical 
approaches to the diverse needs of the hospitality industry. In support of this goal, faculty members are selected 
from appropriate academic disciplines and from the professional community on the basis of their knowledge, 
teaching skills, and practical experience. The field is among the largest in the world, is one of the fastest growing 
industry sectors, and has a global reach. The College has attracted a significant proportion of its student body from 
outside of the United States. At the same time, the dean and some College donors have recognized that 
opportunities for serving significant new markets could be seized by reaching beyond the campus classroom by 
developing and offering an effective distance learning program. 



The College has been offering distance learning courses by Instructional Television (ITV) in the Houston area with 
programs broadcast over cable television and through a tape purchase. Tape purchase permits previous y prepare 
classes to be distributed to students without access to the cable, or for students being offered the course at a different 
time In both cases, instructors interact with students in an asynchronous mode for discussion, questions 
examinations, projects and other assignments. Although the ITV offerings have been popular with students, and 
have been shown to be effective, the College wished to expand its offerings and to introduce additional program 
offerings and flexibility by making courses and degree programs available online, to be delivered through the 
Internet. The first program selected for full online delivery was the College’s Master of Hospitality Management 
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program. This degree offering is particularly aimed at persons who are at work in the hospitality industry, who have 
an earned bachelors degree, and who wish to acquire additional management knowledge and skills to progress to 
more responsible and remunerative jobs in hotels, food service, or other hospitality management sectors. These 
students would seem to be geographically dispersed, unwilling to leave their current jobs to pursue an advanced 
degree, and relatively facile in the use of computer and information technology. Thus an online degree seems to 
effectively meet these needs. 

The Challenge 

As the College sought to develop an online program, many types of questions arose. Those questions related to the 
appropriate technology to be used, the design considerations in course development, the ways in which the course 
are to be taught, the kinds of subject matter and content, and even the fundamental question as to whether 
“traditional” delivery methods and modes were to be used. To begin to answer some of those questions, and to begin 
to develop an overall approach to the program concept, assistance was sought from several viewpoints: current 
students, industry leaders, and members of College business advisory groups. Focus groups were conducted with 
industry experts, and with groups of students who had and had not participated in College distance learning 
programs. The student-centered focus groups helped to identify features to enhance the learning processes and the 
industry-centered groups helped to focus on curricular and program content issues. 

The Delphi Process 

With the data at hand, it also became evident that a clear consensus and agreement on program outcomes among 
faculty members was required. The faculty of the College represent a very diverse group with respect to teaching 
areas, technology expertise, comfort with different teaching and learning strategies, and educational background. 

The problem was to motivate the faculty to agree on goals for the program as a whole, courses within that program 
and competencies for the students. Consensus by all faculty was the goal, because consensus is more likely to result 
in implementation than if resolutions are imposed (Likert, 1967; Ouchi, 1981; Peters & Waterman, 1982). In 
addition, different faculty held varying definitions of “online,” “distance,” “distributed,” “learner-centered” and 
other fundamental concepts. To clarify those issues, and to seek consensus among faculty members, a Delphi 
process was utilized. 

The Delphi process may be characterized as a method for structuring a group communication procedure so that the 
process is effective in allowing a group of individuals, as a whole, to deal with a complex problem (Linstone, 1975, 
p.3). As with the oracle at Delphi, an intermediary, in most cases a questionnaire or survey, is used to maintain the 
anonymity of responses and avoid the bias of dominant individuals. The Delphi process achieves statistically 
derived group consensus by having a group, in this case, the faculty and administrators of the College, complete a 
series of questionnaires (Helmer, 1966). Each series, or round as they are called, provides feedback from the 
previous one. There are usually three to four rounds that culminate either in consensus for each item or no consensus 
and stability. 

A modified Delphi Technique was chosen as the research methodology for this study because of several factors. It 
was selected because first, the problem of determining goals for a program does not lend itself to precise analytical 
techniques but can benefit from the, subjective judgments of a group of experts, in this case the faculty and 
administrators of the College. Answers can be vague, subject to many interpretations, and thus cannot be found 
simply by sifting through relevant information (Pill, 1971; Helmer, 1966; Linstone & Turoff, 1975). 

Second, the Delphi Technique is a common method to use in situations where goals are to be identified (Dixon & 
Henkelman, 1991). In addition, Finch (1993) states that "the Delphi Technique enables experts to speculate 
individually and then reach consensus collectively regarding the content necessary to prepare workers, even in areas 
where no workers exist at the present time" (p. 156). Third, the group of experts needed to contribute their opinions 
to this study has constraints on their relationships that are imposed by tenure, academic rank, and perceived 
relationships in regard to power. In addition this expert group has limitations on meeting times, and interacting in a 
face-to-face exchange may not be the best and most productive means of gathering input (Helmer, 1966; Linstone & 
Turoff, 1975). The Delphi Technique represented the most economical and cost efficient method of soliciting expert 



opinion and arriving at group consensus on the issues since participants may submit their answers at their 
convenience without assembling the entire group at a given time. 

Fourth, since the responses of the participants in a Delphi survey are anonymous to the others, there are no bias of 
leadership influence, face-to-face confrontation or group dynamics (Smith, 1976). This anonymity offers a distinct 
advantage for all respondents; it can offset a domineering personality, fear of losing face by bringing up original 
ideas, and difficulties in publicly contradicting individuals of higher rank (Dailey 1990) and enables respondents the 
opportunity to be more thoughtful and creative (Finch, 1993). Each participant has an equal chance to answer. In 
addition, it is expected that the procedure itself will provide the participants with the opportunity to clarify their own 
thinking. Therefore the Delphi Technique appeared to be the best way of structuring communication among the 
members of an expert group in order to create what Linstone (1975) called a "collective human intelligence" (p.5) 
which includes attitudes and feelings. 

Methodology 

The Delphi technique is a procedure that involves the repeated or iterative consulting with a number of informed 
persons, usually experts in the field, asking them first to individually generate and subsequently assess a specific 
number of statements (Smith, 1976). The generation of statements is termed Round One. The responses of all 
participants are then assembled and redistributed to each participant for Round Two, asking them to rank each using 
a likert scale. Responses are recirculated to the participants in Round Three, and the group is asked to consider their 
responses in light of the total responses of the group. In this way the group moves toward consensus on issues until 
the desired outcome has been achieved. The actual Delphi technique may vary considerably, depending on the 
situation, but the primary utility is, essentially, that it systematically produces a well-considered and informed 
consensus of the perceptions and judgments of a plurality of informed persons (Helmer, 1966). 

Statistical Interpretation 

Because the goal of Delphi surveys is to reach a consensus of opinion, statistics that reflect convergence around the 
median are used (Holden and Wedman, 1993). In most Delphis, consensus is assumed to have been achieved when a 
certain percentage of the votes fall within a prescribed range (Scheibe et al, 1975). In order to determine consensus 
in this study, the quartile deviation was used. The quartile deviation is defined as one-half the interquartile range or 
the difference between the 25th and 75th percentile in a frequency distribution. 

For this study, the measure of high consensus developed by Faherty (1979) was used. Statements that received a 
quartile deviation <.60 were considered to have achieved high consensus. Statements that received >.60 and <1.00 
were defined as having moderate consensus, and statements that received >1.00 were considered having low 
consensus. The premise underlying the Delphi technique is that by the final round of the survey, the quartile ranges 
of the scores for each statement will have diminished to a smaller interval. It is usually asserted that this statistical 
phenomena indicates a more concise agreement among the participants (Faherty, 1979). The iterative process of the 
Delphi survey was terminated when consensus of 95% of the panel’s responses had been reached. 

A goal considered to be "critical" among the participants was one that received a mean score of 5.5 or higher on a 
six-point scale by the final round. A goal rated "very important" was one that received a mean score between 4.5 and 

5.49. A goal rated "important" had a mean that fell between 3.5 and 4.49. Goals that were rated between 2.5 and 
3.49 would be rated "somewhat important." Goals were rated "slightly important" if the mean fell between 1 .5 and 

2.49. Finally any goals whose mean fell between 1.0 to 1.49 were rated "not important." 

Round One 

Participants were asked to brainstorm responses to the following question, "What are the desired goals for the 
Online Masters Program for the College?" and email their responses. Ten faculty members participated in this round 
and generated 78 goals for the program. Duplicates of these 78 responses were discarded, and some were combined 
to generate three broad areas of goals: program goals, course goals, and student competencies. There were 22 goals 
for the program generated, 20goals for courses generated, and 23 goals for graduates of the program generated. 

Round Two 

In Round Two, participants were asked to rate each goal on the following Likert-like scale: 
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l=Not important, 2=Slightly important, 3=Somewhat Important, 4=Important, 5=Very Important, 6=Critical 

A six-point likert scale was selected because a middle rating was not desired. The participants were asked to mark 
the rating of each goal that they felt best indicates their analysis of importance. Space was provided at the end of 
each section of the survey for the addition of goals which participants feel might have been omitted. The fact that 
very few of the participants chose to suggest any additional goals suggests that the first round survey captured the 
data accurately. 

An example of Delphi items for each section of Round Two is presented below: 

The Online Masters Degree program should: 

Present materials in an integrated manner from course to course and across topical areas. 
l=Not important, 2=Slightly important, 3=Somewhat Important, 4=Important, 5=Very Important, 

6=C ritical 



Individual courses in the online Masters Program should: 

Be targeted to mid-career hospitality management professionals. 

l=Not important, 2=Slightly important, 3=Somewhat Important, 4=Important, 5=Very Important, 
6=C ritical 



At the completion of the Online Masters Program, graduates should: 

Demonstrate competencies in research methods, statistical analysis and writing skills. 

l=Not important, 2=Slightly important, 3=Somewhat Important, 4=Important, 5=Very Important, 

6=Critical 



The Round Two survey was given to the participants at a faculty retreat with the instructions to return the completed 
survey in an envelope as they left. Fifteen faculty members completed Round 2. 



Round Three 

The results of Round Two were quantified into median and quartile range scores, and two pieces of information for 
each goal were provided for the Round Three survey: the interquartile range of ratings for the group and the 
individual participant's rating. Instructions were included with the Round Three survey explaining that the responses 
of the majority of the participants were denoted by placing numerals in parentheses to indicate the interquartile 
range, which is both a value and an interval on the continuum. The rating that the individual panel member gave that 
goal was indicated by an X over the number. An example of this is presented below. 



The Online Masters Degree Program should: 
x 

r 2 3 (4 5) 6 2. Permit learners to move in cohorts 

Individual rating Interquartile range of panel 



Round Three surveys were sent to all faculty members and administrators even if they did not participate in Round 
Two. Participants were be asked again to rate each goal, noting the item’s interquartile range in relation to the panel 
member’s previous rating. No additional goals were added after Round Two. Panel members were allowed a time 
limit of 7 days to respond to the Round Three survey. Panel members who did not return their surveys by the 4th day 
were sent personal reminders. 



Results of Round Three 

Responses from Round Three were analyzed and all statistics, including stability measurements, were computed. 
The premise underlying the Delphi method is that by the final round of the study, the quartile ranges of the scores 
for each statement will have diminished to a smaller interval. It is usually asserted that this statistical phenomenon 
indicates a more concise agreement among the panelists that was true in this Delphi study as well. Of the 67 goals, 
64 achieved high consensus by the end of Round Three. The list of goals was prepared with each goal statement 
listed in order of importance as determined by the mean. The ten highest ranked goals are shown in Table 1 
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Round 3 
Mean 


Type of 

Goa! 


Statement 


5.47 


Program 


The Online Masters Degree Program should be world-class, high quality, and 
recognized globally. 


5.31 


Course 


Individual courses in the Online Masters Degree program should be learner- 
centered. 


5.29 


Course 


Individual courses in the Online Masters Degree program should be rich content 
in which is presented effectively. 


5.19 


Program 


The Online Masters Degree Program should have readily available academic 
support with timely responses. 


5.19 


Program 


The Online Masters Degree Program should foster, encourage, practice, vision, 
and apply principles of critical thinking, creativity, and imagination. 


5.18 


Program 


The Online Masters Degree Program should be economically viable 


5.18 


Program 


The Online Masters Degree Program should have a flexible program design to 
permit frequent revision. 


5.18 


Course 


Individual courses in the Online Masters Degree program should contain 
adequate intellectual property protections. 


5.18 


Course 


Individual courses in the Online Masters Degree program should present an 
interesting and satisfying learning experience. 


5.18 


Course 


Individual courses in the Online Masters Degree program should be relevant to 
student needs and goals, and to job behavior and task skills. 



Table 1: Ten Highest Ranked Statements at the End of Round 3 

At the end of Round Three, there were no statements that received a mean score of 5.5 or higher which would 

indicate a "critical" rating, although one was very close: The Online Masters Degree Program should be world-class, 
high quality, and recognized globally ( M = 5.47). Fifty-one goals were rated "very important"; these received a 
mean score between 4.5 and 5.49. Thirteen goals were rated "important" and had a mean that fell between 3.5 and 

4.49. Only two goals were rated between 2.5 and 3.49 and were rated "somewhat important." These were: (1) The 

Online Masters Degree Program should provide a 100 percent placement of graduates. ( M = 3.35), and (2) 
Individual courses in the Online Masters Degree program should be taught by teaching teams. (M= 3.47) Only one 
goal was rated "slightly important": The Online Masters Degree Program should not contain a residency requirement 
as a major consideration. (M= 2.18) 

Discussion 

The Delphi process appeared to function as planned for this study. Even thought the faculty had little or no previous 
experience with Delphi methodology, the surveys were completed correctly and in the allotted timeframe. The three 
rounds of the study produced convergent results that appear reliable. Between Delphi Rounds Two and Three, the 
percentage of statements that reached high consensus increased dramatically, going from 41 statements in Round 
Two to 64 statements in Round Three. Of this number, 13 statements that previously had low consensus and 9 
statements that had moderate consensus moved to high consensus. This change is illustrated in Table 2. There were 
only two goals that had low consensus, and both had an interquartile deviation of 1: (1) The Online Masters Degree 
Program should not contain a residency requirement as a major consideration, ( M = 2.18), and (2) The Online 
Masters Degree Program should have a face-to-face component, in Houston or elsewhere. (M= 3.71) 

Based on the results of this study, the College has data that provides a clear consensus among faculty members on 
most of the program, course, and student goals. It is one tool that will be used to inform a comprehensive needs 
assessment model, supplement other types of feedback, and can be used as planning guide by College 
administration. Although the implications of this study cannot be generalized to other colleges, it appears that the 
Delphi process was useful in capturing and refining a clear picture of current faculty viewpoints and perspectives, 
and provides a useful method for eliciting faculty opinions and encouraging consensus. 
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Program Goals (22 Total) 



Round 2 
Round 3 


High Consensus 
15 
20 


Moderate Consensus 
0 
0 


Low Consensus 
7 
2 


Course Goals (20 total) 


Round 2 
Round 3 


High Consensus 
11 
20 


Moderate Consensus 
4 
0 


Low Consensus 
5 
0 


Student Goals (23 


- Round Two; 25 - Round Three) 




Round 2 
Round 3 


High Consensus 
15 
24 


Moderate Consensus 
5 
1 


Low Consensus 
3 
0 


Total for All Goals (65 - Round Two; 67 - 


Round Three) 




Round 2 
Round 3 


High Consensus 
41 
64 


Moderate Consensus 
9 
1 


Low Consensus 
15 
2 



Table 2: Consensus on Goals Reached at the End of Round 2 and Round 3 
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